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TO BOLSTER SCHOOL REFORM EFFORTS IN LOW- 

performing schools, the Tennessee Depart- 
ment of Education established the 
Tennessee Exemplary Educators program 
in 2001 . Exemplary Educators are recently 
retired educators who are selected to assist 
low-performing schools to improve student 
performance. In general, student achieve- 
ment has increased in these schools. Re- 
ports prepared by Edvantia for the 
Tennessee Department of Education de- 
scribe program features and student 
achievement in participating schools. 

Exemplary Educators. Exemplary 
Educators work with schools to help them 
identify critical needs and implement 
changes that will result in higher student 
achievement. Currently, 95 of these school 
performance coaches work in more than 
155 High Priority schools and 22 High Pri- 
ority school districts in Tennessee. Seven 
cohorts of Exemplary Educators have been 
selected, beginning in November 2000. 
Each Exemplary Educator was selected 
through a rigorous application and inter- 
view process and receives more than 200 
hours of intensive professional develop- 
ment each year from Edvantia. 

Most Exemplary Educators hold 
master’s degrees, and the majority have 
education beyond a master’s degree. They 
have an average of 34 years of experience 
in the field of education and have held a 
range of positions in schools, districts, or 
other educational venues. 

Although Exemplary Educators ap- 
proach their work with a common orienta- 
tion, Edvantia researchers looking across 



schools have discerned no pattern linking 
specific Exemplary Educator characteristics 
or activities with schools’ making adequate 
yearly progress. It appears that they provide 
highly individualized assistance to each 
school based on the school’s needs and 
strengths. 

Student achievement. Exemplary Edu- 
cators have generally been assigned to pro- 
vide assistance to schools that serve 
predominantly economically disadvantaged 
students. In general, student achievement has 
increased in schools where Exemplary Edu- 
cators have provided assistance. The size of 
the increase varies from school to school, and 
increases have not occurred in some schools. 
However, the gains in schools assisted by 
Exemplary Educators overall are comparable 
to or greater than gains in typical and high- 
performing Tennessee schools. Schools as- 
sisted by Exemplary Educators for two years 
have recorded larger gains on average than 
schools that have been assisted for one year. 

Edvantia researchers made these obser- 
vations by comparing student achievement 
levels in mathematics and language arts prior 
to and subsequent to the introduction of 
(continued on page 2) 
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Exemplary Educators. First, they calculated 
an average on each achievement indicator for 
three years prior to the introduction of the 
program (1999, 2000, and 2001). Second, 
student achievement on the same indicator 
was computed for the years subsequent to 
the introduction of the program (2002 and 
2003). Third, a gain (or loss) score was de- 
termined by calculating the difference be- 
tween these two average scores. 

Gain/loss scores were then compared 
with gain/loss scores for other Tennessee 
schools observed over the same period of 
time — Title I schools that rolled off the state 
list of schools needing assistance and ad- 
ditional comparison groups identified as 
“typical” or “high performing” according 
to reading and math Normal Curve Equiva- 
lent scores (for elementary schools) and 



A STUDY OF A CULTURALLY RESPONSIVE 

teaching intervention in four West Vir- 
ginia schools indicates that the approach 
can have positive effects on teacher and 
student classroom behaviors — including 
time on task. However, findings reported 
by the Appalachia Educational Labora- 
tory at Edvantia indicate that teachers 
need intensive training and ongoing sup- 
port if schools are to realize the full ben- 
efits of the intervention. 

Culturally responsive teaching is based 
on the idea that culture is central to student 
learning and teachers can be more effec- 
tive when they tap into students’ cultural 
characteristics, experiences, and perspec- 
tives. The Education Alliance at Brown 
University has examined research related 
to this concept and identified nine prin- 
ciples that guide culturally responsive 
teaching. The intervention examined in the 



reading and math ACT scores (for high 
schools). 

The researchers observe that, in gen- 
eral, the longer an Exemplary Educator as- 
sisted an elementary school, the greater the 
gains in student achievement — although the 
relationship between length of assistance 
and student achievement was not as clear 
in high schools. The sustainability of stu- 
dent achievement increases in schools that 
receive assistance from Exemplary Educa- 
tors has yet to be determined. 

Notable insights. The researchers note 
that the amounts of increase in student 
achievement vary from school to school, 
and increases in student achievement have 
not occurred in some schools. Causes of 
variability and factors contributing to the 
lack of success need to be identified, and 
procedures to counteract factors need to be 
implemented and evaluated. 



Edvantia study incorporated these prin- 
ciples: communication of high expecta- 
tions; active teaching methods; teacher as 
facilitator; attention to culturally and lin- 
guistically diverse students; cultural sensi- 
tivity; reshaping the curriculum; culturally 
mediated instruction; student-controlled 
classroom discourse; small-group instruc- 
tion and academically related discourse. 

Eight schools in the same district par- 
ticipated in the 2003-2004 study, four as 
pilot schools for the intervention and four 
as comparison schools. The researchers 
used a quasi-experimental research design; 
that is, there was a control group, but teach- 
ers were not randomly assigned to full treat- 
ment, partial treatment, or control groups. 
The comparison schools were selected be- 
cause their demographics and achievement 
levels most closely matched those of the 
pilot schools. (In the pilot schools, the per- 



Dr. James Craig, 
director of research at Edvantia, 
may be reached at 
Jim. Craig@edvantia.org. 



Source of Research 

James Craig, Aaron Butler, and Steven 
Moats, An Analysis of the Attributes and 
Work of Exemplary Educators: Summary 
of Findings, 2005; James Craig, Aaron 
Butler, Lori Parker, Kristine Chadwick, 
Chandra Wood, and James Garrett, Status 
Report Review and Analysis: Summary of 
Findings, 2005; Gains in Achievement for 
High Priority Elementary and Middle 
Schools, School Year 2003-2004: 
Preliminary Report of Findings, 2005; An 
Initial Report on the Effects of Tennessee’s 
Exemplary Educators on High Priority 
Schools, 2004; Key Features of 
Tennessee's Exemplary Educators, 2004. 
All reports were prepared by Edvantia 
(formerly AEL) for the Tennessee 
Department of Education. 



centages of students qualifying for free or 
reduced-price lunch were 33%, 60%, 67%, 
and 78% and the percentages of students 
who were African American were, respec- 
tively, 27%, 35%, 34%, and 76%.) 

The researchers note that none of the 
comparison schools had been designated by 
the state as low performing, while three of 
the four pilot schools had received this des- 
ignation. They also state that a randomized 
experiment was not possible for several rea- 
sons — for instance, participating schools 
were selected by the superintendent and 
pilot school team members were selected 
by the principals. 

The full treatment group consisted of 
22 teachers at the pilot schools (two el- 
ementary, one middle, and one high) who 
participated in professional development 
sessions, attended twice-a-month meetings 
on implementing the professional develop- 
ment, and received ongoing technical as- 
sistance. One subgroup of pilot team 
teachers taught culturally responsive cur- 
riculum units; another subgroup did not. 
The partial treatment group consisted of 
158 teachers from the pilot schools who did 
not participate in professional development 
sessions, attend meetings, or receive assis- 
tance. One subgroup taught culturally re- 
sponsive curriculum units, and another 
(continued on page 3) 
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(continued from page 3) 
subgroup did not. The comparison group 
consisted of 128 teachers in four other 
schools (two elementary, one middle, and 
one high) who received no training or cul- 
turally responsive materials. 

Data collection methods included ad- 
ministration of various Edvantia-developed 
paper-and-pencil instruments, classroom ob- 
servations, and analysis of student results on 
the statewide achievement test. Focus groups 
and interviews with project participants pro- 
vided contextual data. Multiple perspectives 
were sought in order to triangulate data and 
increase the validity of findings. 

Edvantia researchers found that 
teachers in the full treatment group who 
taught a culturally responsive instruc- 
tional unit developed by Edvantia created 
a more positive classroom learning en- 
vironment and demonstrated better use 
of class time than teachers in the partial 
treatment and control groups. These 



In the spring of 2004, the Tennessee De- 
partment of Education began delivering an 
online professional development course to 
56 elementary schools with Reading First 
grants. Researchers at the CNA Corpora- 
tion collaborated with the course develop- 
ers and implementers at Edvantia to assess 
the program’s cost effectiveness and its ef- 
fects on teacher knowledge in literacy in- 
struction. Their analysis suggests that the 
online course was cost effective and re- 
sulted in positive effects on teacher knowl- 
edge. 

The 16-week course. Assessment and 
Intervention in a Comprehensive Literacy 
Classroom, was developed by Edvantia to 
help K-3 teachers, K-12 special education 
teachers, and building-level administrators 
to incorporate formal and informal assess- 
ments into reading instruction and to use 



teachers added the equivalent of 16.11 
days of instructional time over the course 
of a year by keeping themselves and stu- 
dents on task over 90% of the time, 
which was more than other teachers. 

These teachers also demonstrated a 
significantly higher quality of instruction 
and had more success engaging students in 
interactive instruction than teachers in other 
classrooms studied. At the close of the 
project, these teachers reported having a 
greater understanding of the causes of 
achievement gaps and the importance of 
culturally responsive teaching strategies in 
closing achievement gaps. 

Researchers found that in pilot schools, 
students’ perceptions of (1) belonging to 
their school community, (2) their ability to 
do well academically, and (3) their fami- 
lies’ expectations of them all improved. 

Notable insights. The study showed 
that the intervention was more likely to 
have a positive effect on teachers’ beliefs. 



the results of those assessments to guide 
instruction. The course includes five mod- 
ules: (1) a conceptual model of a compre- 
hensive literacy classroom in which all 
children can learn to read well by the end 
of the third grade; (2) the five essential el- 
ements of reading instruction; (3) the Ten- 
nessee assessment system and its 
assessment types (screening, diagnosis, 
outcome, and progress monitoring); (4) as- 
sessment instruments, implementation 
steps, and practice opportunities; and (5) 
use of assessment data to guide planning, 
grouping for instruction, and selecting in- 
structional strategies. 

An examination of feedback from four 
sources (online evaluations, threaded dis- 
cussions among course participants, focus 
group surveys, and discussions with lit- 
eracy leaders from the participating 



perceptions, and behaviors when teachers 
received intensive training, experience in 
teaching culturally responsive curriculum 
units, and ongoing support from a skilled 
facilitator. For maximum benefit, Edvantia 
researchers also recommend schoolwide 
implementation. 

Research and evaluation specialist Georgia 
Hughes, lead author for the report, can be 
reached at Georgia.Hughes@edvantia.org. 
Marian Keyes, a research and development 
specialist who works with schools and 
community groups to close achievement gaps, 
may be reached at 
Marian. Keyes@edvantia. org. 

Source of Research 

A team of 10 researchers authored the original 
research report, Effects of a Culturally 
Responsive Teaching Project on Teachers and 
Students in Selected Kanawha County, WV, 
Schools, 2005. An executive summary is 
available from Edvantia at 
www.edvantia.org/publications/pdf/ 
MAACKExecutiveSummary.pdf. 



schools) found high levels of satisfaction 
with the content of the training. Statistical 
comparisons of teacher knowledge before 
the program began and after the comple- 
tion of each module found significant im- 
provements in each area. Data showed that, 
overall, average scores increased 21 per- 
centage points. 

Analysis of collected data indicated 
that introducing the online professional 
development program in the middle of a 
school year was difficult for some partici- 
pants. As a result, the Tennessee Depart- 
ment of Education decided to introduce the 
program at the beginning of the school year 
thereafter. The researchers observed tech- 
nology infrastructure difficulties at the 
school level, but most of these problems 
were addressed during the course of the 
training. 

The researchers also conducted a cost 
analysis of development and delivery of the 
online program and compared the cost of 
delivering the same materials through face- 
to-face workshops. Costs were considered 
from the perspectives of the developer, the 
state (which paid for course development), 
and the schools that purchased course seats 
(continued on page 4) 
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(continued from page 3) 

for their teachers. Taking into account all 
development and delivery costs, they esti- 
mated that the provider spent $24.81 per 
course seat on the creation and delivery of 
the training, a cost that will decrease as the 
enrollment in the course increases. This cost 
is less than half that of a similar face-to- 
face course, which the researchers esti- 
mated at $70 per course seat. 

Researchers noted that they had hoped 
to examine student outcomes in the 56 
Reading First schools relative to student 



Eleven members of The Renaissance 
Group, a 10-state consortium of 36 colleges 
and universities noted for their teacher edu- 
cation programs, are using a Title II grant 
to develop systems to measure and improve 
teacher candidates’ ability to facilitate P- 
12 student learning. A 2003 study con- 
cluded that two components of the Title II 
Renaissance Partnership for Improving 
Teacher Quality Project (teacher work 
samples and development of an account- 
ability system) have great potential for im- 
proving the teaching skills of graduates and 
effecting institutional improvement. Four 
participating universities have demon- 
strated progress that is above the group 
average on both components. A follow-up 
evaluation revealed that internal and exter- 
nal factors contributed to their progress. 

The key component of The Renais- 
sance Group’s effort is the creation of a 
“circle of feedback” among teacher educa- 
tors, program graduates (new and in-ser- 
vice teachers), and children’s achievement 
in the classroom. Teacher work samples, 
classroom observations, and other mea- 
sures assess the performance of individual 



outcomes in a sample of matched compari- 
son schools. However, due to limitations 
in the public availability of the 2005 stu- 
dent achievement data (TCAP and 
TVAAS), they have not yet been able to 
fully explore the relationship between 
teachers’ online professional development 
and its possible impact on their students’ 
academic achievement. 

Notable insights. Limitations in the 
availability of student achievement data un- 
derscore the difficulty of quantifying the 
relationship between professional develop- 
ment and student achievement. 



teacher-student pairs relative to national 
and state teaching standards. 

Teacher work samples are exhibits of 
teaching performance (e.g., written narra- 
tives, assessment instruments, student 
work) that provide direct evidence of a 
teacher’s ability to design and teach stan- 
dards-based instruction, assess student 
learning, reflect on the teaching/learning 
process, and help all students learn. Work 
samples give both P-12 and university 
teachers direct evidence of whether their 
teaching is improving student learning on 
professional standards. This assessment 
method, known as the Teacher Work 
Sample Methodology, was developed at 
Western Oregon University and was modi- 
fied by participating higher education in- 
stitutions. 

Edvantia researchers used a case study 
approach to identify factors contributing to 
above-average progress in instituting 
teacher work samples and establishing ac- 
countability systems. During two-day site 
visits conducted at each of the four above- 
average universities in the fall of 2004, 65 
individual and group interview sessions 



Linda Cavalluzzo, a senior research analyst at 
CNA Corporation, can be reached at 
cavallul@cna.org. John Ross, a senior 
research and development specialist in 
instructional technology at Edvantia, can be 
reached at John.Ross@edvantia.org. 

Source of Research 

John Ross, Carol Thigpin, Linda Cavalluzzo, 
Joseph Guzman, and Leslie Patterson, A Study 
of Perceptions and Changes in Knowledge, 
Practice, and Student Achievement as a Result 
of AEL’s Online Professional Development in 
Reading in Tennessee, 2004. Linda Cavalluzo, 
David Lopez, and John Ross with Miguel 
Martinez, Cost-Effectiveness Analysis of Online 
Professional Development, 2005. 



were conducted with a total of 209 people, 
including 18 university administrators and 
project coordinators, 79 faculty members 
and technology staff, 85 teacher candidates 
and recent graduates, and 27 cooperating 
K-12 school and district personnel. 
Edvantia also examined relevant materials 
and electronic data management systems. 

Teacher work samples. The research- 
ers found that the concept of teacher work 
samples as both a process and a product 
has become embedded in the cultures of 
the four universities and enjoys strong com- 
mitment from university leadership. Aware- 
ness and training sessions were offered for 
various stakeholder groups; also offered 
was formal training on scoring teacher 
work samples. At two universities, univer- 
sity-wide faculty councils were instrumen- 
tal in making teacher work samples 
mandatory for all student teachers. 

Accountability systems. All four uni- 
versities have strong administrative support 
and commitment for developing and using 
a data management system. Routine report- 
ing of data is becoming standard within 
each university. The universities built sys- 
tems that are Web based and include both 
custom and commercial software programs. 

Across the four universities, three key 
factors contributed to the institutionaliza- 
tion of accountability systems: external 
pressures, strong leadership, and resources. 
External factors included the need to de- 
velop data-based accountability systems for 
the schools’ National Council for Accredi- 
(continued on page 5) 
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(continued from page 4) 
tation of Teacher Education accreditation 
visits and the growing pressure on K-12 
schools to implement data-driven instruc- 
tional decision making. Strong leadership 
was evident in administrative support for 
system development and for a team ap- 
proach to designing, developing, and imple- 
menting accountability systems. The 
Renaissance grant, which covered staffing 
and technology, provided vital support. 

Improving teacher quality. Case stud- 
ies at the four universities revealed several 
ways in which teacher work samples and 
accountability systems are improving 
teacher education and, thereby, improving 
teacher quality. (1) Teacher work samples, 
by design, shift the focus from textbook- 
driven instruction to data-driven instruc- 
tion. (2) Faculty members’ instructional 
successes and weaknesses become more 



Between 2002 and 2005, the Appalachia 
Educational Laboratory at Edvantia worked 
in partnership with the Virginia Department 
of Education and a low-performing urban 
school district to design and test an inter- 
vention called the Partnership for Achiev- 
ing Successful Schools (PA+SS IV). This 
initiative called for a system of external 
facilitation focused on data-based instruc- 
tional decision making and strategic im- 
provement planning. An evaluation 
prepared by the CNA Corporation (CNAC) 
found that implementation proceeded 
slowly at first due to mistrust of the inter- 
vention among school leaders, lack of sup- 
port from the school board, and frequent 
turnover among professional staff (approxi- 
mately 50% annually). It appears, however, 
that the external facilitators (called “school 
improvement specialists”) were able to es- 
tablish working relationships with school 
staff and to move forward with data analy- 
sis, school improvement planning, and cur- 



evident during reviews of their students’ 
completed work samples. (3) The univer- 
sities have used their experiences with 
teacher work samples to modify their cur- 
ricula. (4) The teacher work samples and 
the accountability system have increased 
faculty conversations and cooperation 
within and across departments. 

Edvantia researchers note that the uni- 
versities studied have completed the first 
steps necessary to investigating the relation- 
ship between teacher-graduate performance 
and P-12 learning. 

Notable insights. It is clear that the 
success of the four universities studied 
would have been much more difficult, if 
not impossible, to achieve without the fi- 
nancial support of the Renaissance grant 
and the synergistic information sharing that 
the Partnership project promoted among its 
participants. 



riculum alignment. Available data show 
improvements in schools’ professional 
learning culture and an overall increase in 
pass rates on Virginia’s state exams for the 
majority of students at all grade levels. 

The goal of the PA+SS IV intervention 
was to help the district schools meet all cri- 
teria for state accreditation. Improvement 
specialists worked in each of the district’s 
10 schools, and a specialist was also placed 
in the central office. Each specialist was an 
experienced educator who had successfully 
led a similar school in the past. All received 
training from Edvantia in the use of data for 
decision making, school improvement plan- 
ning, curriculum alignment, monitoring in- 
struction, and monitoring student 
achievement. They also received in-depth 
training in facilitation skills. 

The specialists’ role was to support 
schools by guiding school personnel in as- 
sessing needs and developing strategic im- 
provement plans. School improvement 



Kimberly S. Cowley 

(Kimberiy.Cowiey@edvantia.org) is a research 
and evaluation specialist at Edvantia. Dr. 
Merrill L. Meehan, Edvantia’s director of 
evaluation, may be reached at 
Merrill. Meehan@edvantia. org. 



Source of Research 

Kimberly S. Cowley, Susan Voelkel, and Nicole 
L. Finch, Teacher Work Samples and 
Accountability Systems: An In-Depth Study of 
Successful Implementation at Four Universities 
Participating in the Title II Renaissance 
Partnership for Improving Teacher Quality 
Project, 2005. Kimberly S. Cowley, Nicole L. 
Finch, Merrill L. Meehan, and David Holdzkom, 
Formative Evaluation of the Title II 
Renaissance Partnership for Improving 
Teacher Quality Project, 2003. The formative 
evaluation received a first-place award from 
the American Educational Research 
Association in the School Evaluation and 
Program Improvement Division’s Outstanding 
Publications Competition in 2004. Information 
about teacher work samples and the AEL (now 
Edvantia) independent evaluation of the 
Renaissance Partnership Project is available 
on The Renaissance Group’s Web site 
(http://fpAjni.edu/itq/). 



specialists were assigned to work on site 
at a school at least one day a week. The 
specialist assigned to the central office ini- 
tially worked 20 hours a week, but the as- 
signment expanded over time to four days 
a week. The specialists were involved in 
creating, implementing, and monitoring 
the school improvement plans. Two major 
thrusts of the specialists’ work were 
(1) training and assisting teachers and ad- 
ministrators in analyzing data for instruc- 
tional planning and (2) aligning the 
curriculum to state standards. 

CNAC evaluators examined the fidel- 
ity with which the PA+SS IV intervention 
was implemented, its effects on the quality 
of school improvement plans, and its im- 
pact on school climate and instructional 
quality. They used a matched comparison 
group design, using school-level data to 
provide a rough initial measure of the 
intervention’s success in improving student 
achievement. 

The evaluators found that implemen- 
tation of the intervention model was 
achieved through the multiyear presence of 
the improvement specialists and the atten- 
(continued on page 6) 
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Improvement Specialists 

(continued from page 5) 
tion that was paid at the building level to 
increasing each school’s capacity for im- 
provement. Analysis of school improve- 
ment plans indicates that the content of all 
school plans matched the PA+SS IV rubric 
indicators, but the evaluators were not al- 
ways able to determine whether schools 
successfully implemented the plans be- 
cause some schools lacked detailed docu- 
mentation. Interviews with school 
improvement specialists and analyses of 
their weekly journal entries indicated that 
principals and central office staff showed 
a great deal of interest and focus in com- 
pleting the plans, but little use was made 
of the completed documents in gauging 
school progress toward defined goals. “The 
challenge,” said the central office school 
improvement specialist, “is to simply make 
the school improvement plan more mean- 
ingful.” 

A comparative analysis with matched 
control schools in another district showed 
that schools in the PA+SS IV district are 
making gains in student achievement that 
approximate or exceed the rate of gain in 
the matched control schools and state av- 
erages. Student achievement data show an 
overall increase in state assessment pass 
rates for the majority of students in the dis- 



trict, and a trend line analysis shows that 
most of these pass rates increased at a 
greater rate following implementation of 
the program. 

Researchers acknowledge that pass 
rates are not an ideal measure of student 
achievement but state that these were the 
only student achievement data available to 
them. 

Staff responses to Edvantia’s Continu- 
ous School Improvement Questionnaire, 
which measures commitment to continu- 
ous learning and improvement, revealed 
improvement on all six measures (shared 
leadership, effective teaching, school-fam- 
ily-community connections, purposeful stu- 
dent assessment, shared goals for learning, 
and learning culture). Responses to 
Edvantia’s Measure of School Capacity for 
Improvement indicate that professional 
staff were less satisfied with their schools’ 
capacity for improvement in 2004 and 2005 
than they were in 2003 (the first year of the 
intervention). The evaluators state that such 
dissatisfaction may represent an “imple- 
mentation dip,” wherein staff become 
aware of how much change is needed for 
successful reform. 

Notable insights. The researchers cau- 
don that because achievement data from 
only two post-intervention years could be 
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In 2001, Edvantia and its partner, Western 
Kentucky University, conducted a compre- 
hensive evaluation of Kentucky’s Extended 
School Services (ESS) program, which was 
established in 1990 as part of the Kentucky 
Education Reform Act. The ESS program 
was designed to address the needs of the 
state’s K-12 students who are at risk of aca- 
demic failure by providing additional in- 
structional time before school, after school, 
and in the summer. Nearly all of Kentucky’s 
1,450 schools offer some type of ESS pro- 
gram. 

The evaluation showed the ESS pro- 



gram to be addressing its stated goals and 
having a positive impact on student moti- 
vation — and to be positively perceived by 
involved stakeholders. (Nearly all teachers 
and coordinators also reported increased 
academic achievement among participants, 
but student achievement data were not 
available for analysis.) 

Evaluators offered several recommen- 
dations for improving the program but 
noted that several would require an in- 
creased level of funding. Recommenda- 
tions included addressing the need to recruit 
and retain qualified staff, enhancing pro- 



compared to the pre-intervention achieve- 
ment data, more time is needed to see 
whether the PA+SS IV intervention will 
have the intended effects on student 
achievement. They recommend that school 
culture be measured beyond the three-year 
intervention period to determine whether 
the schools continue to develop capacity 
for improvement. To improve the likelihood 
of sustaining improvements, they recom- 
mend that the district institutionalize the 
central office school improvement special- 
ist position and call on the staff member at 
each school who received the most train- 
ing in data analysis to provide instruction 
in analyzing data for use in instructional 
planning. 

Joanna Edwards, a research project director at 
the CNA Corporation, can be reached at 
edwardj@cna.org. Kristine Chadwick, a 
research and evaluation specialist at Edvantia, 
can be reached at 
kristine. chadwick@edvantia. org. 



Source of Research 

Joanna Edwards, Karen Smith, Linda Marr, 
Laura Wyshynski, and Kristine Chadwick. 

School Improvement in Petersburg: A 
Comprehensive Three-Year Study of the 
Partnership for Achieving Successful Schools 
Initiative Model TV Intervention (Final 
Evaluation Report), 2005. 



fessional development and providing ac- 
cess to a “best practices” database, expand- 
ing services in terms of hours and/or 
subjects, exploring solutions to transporta- 
tion challenges, investigating ways to re- 
cruit hard-to-reach students, and 
encouraging more involvement of parents 
and students in setting goals for individual 
students. 

The evaluation had two major data 
collection components. First, statewide sur- 
veys were administered to populations of 
district and school ESS coordinators. Sec- 
ond, pairs of trained data collectors con- 
ducted two- to three-day visits to a sample 
of 24 elementary, middle, and high schools. 
Site visit data collection instruments in- 
cluded four different surveys, five interview 
protocols, a three-instrument classroom 
observation system, and an innovation con- 
(continued on page 7) 
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(continued from page 6) 
figuration checklist. Evaluators were struck 
by the “marked consistency and high de- 
gree of corroboration both within and 
among respondent perceptions and data 
collector observations.” 

The evaluators made several observa- 
tions about program strengths, the nature 
of instruction, and student outcomes: 

Program strengths. Coordinators and 
teachers agree that the most critical com- 
ponents for successful implementation are 
strong district- and building-level support. 
Process-linked support for implementation 
included targeting students as early as 
possible, dedicated staff, student transpor- 
tation, collaboration between teachers and 
coordinators, flexible scheduling, low 
teacher-student ratio, and individualized 
instruction. Given that student mobility is 
high throughout the program, a unique 
strength is the program’s fluid and flexible 



In 2004, Edvantia and the National 
Association of State Boards of Education 
(NASBE) initiated an effort to identify suc- 
cessful strategies for recruiting and retain- 
ing highly qualified teachers in rural areas. 
They reviewed non-rural-specific and ru- 
ral-specific research and practice literature, 
surveyed rural superintendents across the 
nation, and conducted case studies of three 
Virginia programs that support teacher re- 
cruitment and retention. 

Generally, the literature shows that the 
problem of teacher shortages varies across 
geography, demography, and subject area. 
The schools that find it hardest to recruit and 
retain highly qualified teachers are those in 
highly urban and rural areas (especially those 
serving minority or low-income students) and 
schools in the Southeast, Southwest, and the 
West. Especially needed are teachers in spe- 



accommodations in identification, referral, 
and enrollment. 

Nature of instruction. The quality of 
instruction was better in regular classrooms, 
but the level of instruction was more often 
appropriate in ESS classrooms, where it 
tended to be student-led and individualized. 
ESS classrooms often reinforced learning 
through the use of games, visual aids, prac- 
tice, additional time, and incentives. 

Student outcomes. Nearly all teachers 
and coordinators indicated increased aca- 
demic achievement among participating 
students, but participating students and their 
parents also reported improved study skills 
and increased motivation to learn. 

Notable insights. The evaluators rec- 
ommended that “it should be clearly com- 
municated to all stakeholders that the 
ESS program, as implemented under cur- 
rent laws and regulations, is not designed 
to be an enrichment program. A clear un- 



cial education, bilingual education, math, and 
science. Edvantia/NASBE survey results and 
case studies amplify these findings and offer 
insights into challenges and promising prac- 
tices in rural teacher recruitment and reten- 
tion. 

Literature review. Rural-specific litera- 
ture identifies four challenges related to 
recruiting and retaining teachers in rural 
areas: (1) lower pay; (2) geographic and 
social isolation; (3) difficult working con- 
ditions, such as having to teach classes in 
multiple subject areas; and (4) NCLB re- 
quirements for highly qualified teachers 
(e.g., many rural teachers will need certifi- 
cation in multiple subject areas, and pro- 
fessional development opportunities can 
sometimes be scarce in rural communities). 
Collectively, these challenges can place 
rural schools and districts at a competitive 



derstanding of the specific nature and pur- 
pose of the statewide program may help 
avoid efforts to shift its focus from strug- 
gling learners to all students.” 

Kimberly S. Cowley is a research and 
evaluation specialist at Edvantia. She may be 
reached at Kimberly.Cowley@edvantia.org. Dr. 

Merrill L. Meehan, Edvantia's director of 
evaluation, may be reached at 
Merrill. Meehan@edvantia. org. 



Source of Research 

Kimberly S. Cowley, Merrill L. Meehan, Nicole 
Finch, Kristine Chadwick, Caitiin Howley, Joy 
Riffle, Jennifer Blake, and Lisa Ermolov, 
Comprehensive Evaluation of the Kentucky 
Extended School Services Program, 2002. 
This report was prepared for the Kentucky 
Department of Education and is available on 
its Web site at www.education.ky.gov/KDE/ 
Administrative+Resources/ 
Data+and+Research/Kentucky+Research/ 
Achievement+Gap+and+At+Risk+Research+.htm. 



disadvantage in attracting and retaining 
well-qualified teachers. 

At the national level, the merits of a 
variety of practices are being examined and 
debated. A look at rural-specific and gen- 
eral literature shows agreement that effec- 
tive recruitment and retention practices 
share four characteristics: They are ( 1) stra- 
tegic, (2) specific to the schools or subject 
areas that are hard to staff, (3) sustained, 
and (4) rooted in the community. 

A survey of literature on state and dis- 
trict strategies revealed five major strate- 
gies being used for recruiting and retaining 
teachers: (1) “grow-your-own” initiatives, 
especially those that help paraprofession- 
als become certified teachers; (2) targeted 
incentives to teach in schools or subject 
areas where the need is greatest; (3) im- 
proved recruitment and hiring practices; (4) 
improved school-level support for teach- 
ers, including formal induction and 
mentoring programs; and (5) use of inter- 
active technologies to meet information and 
professional development needs. 

A review of the research and practice 
literature suggests 14 promising strategies 
for placing high-quality teachers in rural 
classrooms and keeping them there. These 
(continued on page 8) 
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include the strategies mentioned above, 
plus others: base recruitment efforts on data 
analysis, increase the pool of candidates by 
expanding or refining recruitment efforts, 
include all vital partners in collaborative 
efforts, evaluate efforts regularly, encour- 
age universities to customize teacher edu- 
cation programs, include building-level 
staff in the hiring process, improve the 
school’s culture and working conditions, 
involve the community in welcoming new 
teachers, and invest in leadership develop- 
ment. 

National survey. A total of 597 super- 
intendents from a random selection of 1,565 
school districts completed the survey with 
valid data, yielding an overall response rate 
of 38%. The responses of these 597 super- 
intendents reflect the recruiting and reten- 
tion practices of approximately 1,900 
schools serving more than 7 1 8,000 elemen- 
tary, middle school, and high school stu- 
dents from rural areas. 

Survey results echo the literature 
review’s finding that districts located near 
urban areas may have greater advantages 
when compared to districts not located near 
an urban area. Districts located near an ur- 
ban area have more schools within the dis- 
trict and serve more students than those 
districts not located near an urban area. 
However, these same schools also report 
having fewer students qualifying for free 
or reduced lunches, indicating that rural and 
remote schools may have substantially 
more students living in poverty. 

Rural districts reported that their great- 
est challenges in recruiting and retaining 
teachers are geographic and social isola- 
tion as well as being in close proximity to 
higher-paying districts. 

The most frequently cited recruitment 



methods were the use of statewide/local/ 
Internet advertising, personal contacts, and 
networking. Strategies for locating poten- 
tial teachers included involving building- 
level staff in the recruitment and hiring 
process, promoting the advantages of liv- 
ing and teaching in a rural area, and offer- 
ing more competitive salaries. Given the 
resources present in rural districts, the lim- 
ited reliance on the use of targeted incen- 
tives, housing and relocation assistance, 
and collecting relevant data on teacher sup- 
ply and demand is not surprising. 

Teachers who stay in rural districts are 
thought to do so as a result of enjoying their 
position and the overall school and com- 
munity environment, as well as the salary 
and benefits or the stability and conve- 
nience of being in one area. While some 
superintendents indicate that teachers leave 
for personal reasons or to relocate, other 
reasons include poor salary and benefits, 
dissatisfaction with working in a small 
school and living in a rural environment, 
and reduced opportunities. 

Case studies. In 2002, Virginia re- 
ceived a three-year $13.5 million federal 
Teacher Quality Enhancement Grant to 
develop and implement strategies to train 
and retain high-quality teachers. The grant 
funded five recruitment and retention ef- 
forts. Edvantia and NASBE used a case 
study approach to examine three of these 
programs: the Teachers for Tomorrow Pro- 
gram, a precollege recruitment effort; the 
Career Switcher program, aimed at attract- 
ing midcareer teacher candidates; and the 
Teacher Mentoring Pilot Program, which 
supports a variety of new teacher induc- 
tion programs. 

Data were collected via document re- 
views and semi-structured interviews in 
schools in seven rural Virginia school di- 
visions. Documents included Virginia De- 



partment of Education reports and program 
descriptions and materials. Interviews were 
conducted with 51 individuals, including 3 
state directors, 6 division and school ad- 
ministrators, 4 program instructors, 19 stu- 
dents, 6 teacher candidates, 6 mentors, and 
13 beginning teachers. 

Because the programs studied are in 
the early stages of implementation; it is too 
early to judge their overall effect on rural 
teacher recruitment and retention. Prelimi- 
nary data indicate, however, that each pro- 
gram holds promise. The researchers 
concluded that two factors are critical to 
the programs’ continued success: ensuring 
adequate funding and allowing rural school 
districts to adapt programs to meet their 
needs. 

Notable insights. Clearly, highly ru- 
ral schools are at a disadvantage when com- 
peting for teachers in the national pool of 
candidates. Promising strategies for coun- 
tering this disadvantage involve increasing 
the pool of qualified local candidates who 
are already committed to staying in the 
community and supporting their entry and 
early success in rural schools. 

To learn more about the Edvantia/NASBE 
study of rural teacher recruitment and 
retention, contact Edvantia 's director of 
communication, Patricia Cahape Hammer, 
former director of the ERIC Clearinghouse on 
Rural Education and Small Schools. She can 
be reached at Patricia.Hammer@edvantia.org. 

Source of Research 

Patricia Cahape Hammer, Georgia Hughes, 
Carla McClure, Cynthia Reeves, and Dawn 
Salgado, Rural Teacher Recruitment and 
Retention Practices: A Review of the 
Research Literature, National Survey of Rural 
Superintendents, and Case Studies of 
Programs in Virginia, 2005. Published jointly 
by Edvantia and the National Association of 
State Boards of Education. 





Editor’s Spotlight 



Randomized Field Trials: Rosy for Rigor but 
Thorny to Implement 



In previous columns, I have discussed the 
growing “evidence-based policy” move- 
ment in education and its positive effects 
on increasing the rigor and scientific value 
of education research. Consonant with 
these developments, randomized field tri- 
als (RFTs) are gaining saliency as the most 
powerful means of identifying the impacts 
of educational programs on instruction and 
learning. Unlike quasi-experiments, which 
compare pre-existing “treatment” and con- 
trol groups (e.g., schools that use a new pro- 
gram vs. similar schools that do not), 
random experiments eliminate “sampling 
bias” by establishing treatment and control 
group assignments purely on the basis of 
chance. Despite this very important advan- 
tage, the inherent challenge for RFT re- 
searchers is convincing real principals or 
teachers to change what they do education- 
ally, in the interest of furthering education 
science, according to the outcome of a “coin 
flip.” 

Random assignment can take place at 
different levels, such as with individual stu- 
dents, classes (teachers) within schools, or 
schools within a district. In the case of in- 
dividual students, the key question becomes 
whether the treatment and control programs 
(e.g., using textbook vs. computer-based 
exercises) can be implemented simulta- 
neously without arousing parental objec- 
tions or disrupting the normal classroom 
flow. An important precondition is that the 
control treatment represents reasonable (al- 
beit traditional or routine) instruction that 
doesn’t deprive students of needed inter- 
ventions that, in the absence of the research, 
they would otherwise receive. 

The situation, unfortunately, can be- 
come more problematic with complex in- 
terventions that affect the primary 
instructional programs used in a particular 
course or throughout a school. An example 



might be adopting a new math curriculum 
or integrating laptop computer usage with 
science instruction. Prospective RFT par- 
ticipants (teachers and principals) may per- 
ceive a control-group designation as a 
“sentence” to a type of “reform purgatory,” 
requiring them to continue using the very 
strategies that haven’t been successful in 
the past. How can random assignment in 
these situations be made feasible and pal- 
atable? 

A creative solution called “paired 
awards” has been proposed by noted Johns 
Hopkins University researcher Robert 
Slavin. A simplified description of this ap- 
proach is having districts apply for grants 
by identifying pairs of schools (or teachers 
within schools) willing to participate in the 
research. Those randomly selected as con- 
trols would continue using their regular 
programs for one year, but then be eligible 
to adopt the treatment under reduced costs 
or other concessions. In this way, every- 
one stands to benefit by gaining fairly quick 
access to the treatment. But an underlying 
assumption here, with which I strongly dis- 
agree, is that most treatments can be imple- 
mented adequately within the first year to 
demonstrate their true benefits. In my re- 
search experiences, I have frequently seen 
the opposite occur — what some call the 
“implementation dip.” That is, given new 
strategies to understand and put into prac- 
tice, teachers may struggle for a while, with 
short-term negative effects on teaching and 
learning. It would seem ironic if increas- 
ing RFT usage concomitantly caused the 
effectiveness of many potentially valuable 
programs to be systematically underesti- 
mated. 

As a refinement of the above paired- 
awards approach, I advocate that the treat- 
ment-control comparison last a minimum 
of two years. As an incentive for partici- 



pating, all schools (or teachers) in the se- 
lection sample would be offered two years 
of either the treatment or two years of some- 
thing else that is sufficiently attractive. The 
latter could be an alternative new program 
where a comparison study is feasible or 
resources or professional development in 
a different area of need, e.g., classroom 
management, parent involvement, technol- 
ogy resources, or classroom library books. 
While this approach involves more cost, it 
buys essential time for educational pro- 
grams to be tested validly. Knowing more 
confidently what works should pay divi- 
dends many times over the RFT cost. 



Dr. Steven M. Ross has served as the editor 
o/TransFormation, Edvantia's semiannual 
educational research briefing for 
policymakers, since 2001. He also directs the 
Center for Research in Educational Policy at 
The University of Memphis. He can be 
reached at smross@memphis.edu. 








Appalachia Educational Laboratory 
at 



EDVANTIA 



Nonprofit 
Organization 
US Postage Paid 
Permit No. 2560 
Charleston WV 
25301 



P.O. Box 1348 

Charleston, WV 25325-1348 

800.624.9120 • www.edvantia.org 



Edvantia was founded in 1966 as the Appa- 
lachia Educational Laboratory, Inc. (AEL); on 
September 1 , 2005, AEL became Edvantia, Inc. 

The Regional Educational Laboratory for the 
Appalachian region is known as the Appalachia 
Educational Laboratory at Edvantia. 

Edvantia is a nonprofit education research 
and development corporation, founded in 1966, 
that partners with practitioners, education agen- 
cies, publishers, and service providers to im- 
prove learning and advance student success. 

Edvantia provides clients with a range of ser- 
vices, including research, evaluation, profes- 
sional development, and consulting. 

This publication is based on work sponsored 
wholly or in part by the Institute of Education 
Sciences, U.S. Department of Education, under 
contract number ED-01 -CO-001 6. Its contents do 
not necessarily reflect the views of IES, the De- 
partment, or any other U.S. government agency. Edvantia is an equal employment opportunity/affirmative action employer. 



Iff! 



Recommended Reading 



Editor 

Managing Editor 
Policy Analyst 
Research Analyst 

About 
TransFormation 

Edvantia, Inc. (formerly AEL) publishes 
TransFormation twice a year to provide 
policymakers with interpretive summaries of sig- 
nificant reports and rigorous studies about school 
transformation, giving special attention to re- 
search with possible implications for improving 
low-performing schools. It is edited by Dr. Steven 
Ross (smross@memphis.edu), Faudree Professor 
and director of the Center for Research in Educa- 
tional Policy at The University of Memphis. Cur- 
rent and past issues are available online at 
www.edvantia.org/transform. 



Policy briefs from Edvantia (formerly 
AEL) offer research analyses and insights into 
issues of interest to policymakers and educa- 
tion decision makers. All briefs, including those 
described below, are available online. 

• Maximizing the Effectiveness of Online Ac- 
countability Assessments for Students with Dis- 
abilities (2005) by Mary Axelson 

This policy brief describes why and how 
states can use universal design principles to 
maximize the effectiveness of their new assess- 
ments for measuring the knowledge and skills 
of students with disabilities while improving the 
speed and usefulness of their assessment sys- 
tems in general. 

http://www.edvantia.org/publications/ 

PBAssessment.pdf 

• Corrective Action: A Look at State Takeovers 
of Urban and Rural Districts (2005) by Patricia 
Cahape Hammer 

Interest in state takeovers aimed at improv- 
ing student achievement may increase as states 
explore turnaround options for districts identi- 
fied for NCLB-required corrective action. This 



policy brief examines research and data related 
to state takeovers or urban and rural districts 
and suggests measures for increasing the effec- 
tiveness of similar corrective actions. 

http://www.edvantia.org/publications/pdf/ 

PBStateTakeovers.pdf 

• Alternative and Conventional Certification 
for Education Administrators (2004) by Lars G. 
Bjork & James S. Rinehart 

This policy brief looks at the ongoing de- 
bate on the merits of deregulation and alterna- 
tive certification versus strengthening the 
existing systems of certifying administrators. 

http://www.edvantia.org/publications/pdf/ 

PBJune04.pdf 

• High-Quality Teaching: Providing for Rural 
Teachers’ Professional Development (2004) by 
Aimee Howley & Craig B. Howley 

This policy brief examines the need for 
high-quality professional development that en- 
gages rural meanings and is appropriate to the 
structure and dynamics of rural systems. 

http://www.edvantia.org/publications/pdf/ 

PBHigh-QualityTeaching.pdf 
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